Analysis and Synthesis of Sign Language Gestures: from Meaning to Movement Production

The ever growing use of gestures in advanced technologies, such as augmented or virtual reality environments, requires more and more understanding of the different levels of representation of gestures, from meanings to motion involving causal physical and biological phenomenon. This is even more true for gestures used for artistic expression, such as dance or musical gestures, or for communicative gestures such as sign language gestures, both involving high level semiotic and cognitive representations and requiring extreme rapidity, accuracy, and physical engagement with the environment. We first highlight in this paper the tight connection between the high-level and low-level processes, and provide a general outline for motion control of skilled gestures, where different sensory data are used to determine the action to perform in a global action-perception loop. We then propose a methodology for analyzing and synthesizing human expressive and semiotic gestures. We focus on data-driven approaches, using jointly captured motion and annotated videos. Motion capture allows us to find relevant representations that encode the main spatio-temporal characteristics of gestures. Annotations provide a way to introduce significant labels indicating the morphosyntactic nature of elements composing gestures, but also expressive features that are specific to some subject and articulation effects. The nature of the different types of information encoded in and by gestures makes it necessary to store data in two different structures, namely a semantic database for textual annotations, and a raw database for motion capture data. From these coupled databases, it becomes then possible to statistically extract some gesture invariants and to use them for resynthesis. The databases are also exploited for driving our simulations. We believe that data-driven synthesis methods significantly improve the quality and the credibility of the synthesized motion, and also provide significant clues for constraining the produced movements. In the remaining of the paper, we describe several challenges that are involved in the case study of sign language analysis and synthesis. 54